Landsat-5, 7, 8 and 9 collection 2 products are managed by USGS. USGS make Landsat data available via number of services, including:
Surface Reflectance, Surface Temperature and Level-1 (top of atmosphere) products for each of Landsat-5, 7, 8 and 9 are available.
The STAC specification is a common language to describe geospatial information. A STAC API provides a search and selection interface to a catalog of items and files. See https://stacindex.org/catalogs#/ for a list of providers using STAC.
While the STAC specification allows for consistent searching and access to available files, how these files are used and interpreted can still be a challenge or at least specific to each custodian.
This notebook demonstrates how to search, download, visualise and export Landsat and Sentinel-2 satellite imagery.
The Open Data Cube (ODC) records product and scene information in a database and provides tools for tranforming and aggregating scene data into geospatial python xarray "cubes". In this context, the STAC API can replace some parts of the ODC database while providing the core geospatial information required for transforming and aggregating the scene data into cubes.
The ODC odc-stac and odc-geo packages provide the core functionality of reading, tranforming and aggregating files. In particular, the odc-stac library takes a list of STAC items as input and reads these into an xarray cube compatible with ODC functions.
This notebook was adapted from https://github.com/opendatacube/odc-stac/tree/develop/notebooks.
# Minimal packages
import os, sys
from pystac_client import Client
from odc.stac import configure_s3_access, stac_load
# Python packages
import re
import json
import pandas as pd
import numpy as np
from pathlib import Path
# ODC packages
from dea_tools.plotting import display_map, rgb
from datacube.utils import masking
from odc.algo import mask_cleanup, erase_bad, to_f32
from odc.ui import image_aspect
# EASI packages
repo = Path.home() / 'eocsi-hackathon-2022' # No easy way to get repo directory
if repo not in sys.path: sys.path.append(str(repo))
from tools.notebook_utils import heading, xarray_object_size, initialize_dask, localcluster_dashboard
from tools.stac_utils import stac_landsat_assets_df, stac_landsat_flags_to_dc
# Setup
# Does this work stand-alone or require an AWS account?
configure_s3_access(requester_pays=True)
# Optional: use EASI SE Asia caching-proxy service
os.environ["AWS_HTTPS"] = "NO"
os.environ["GDAL_HTTP_PROXY"] = "easi-caching-proxy.caching-proxy:80"
print(f'Will use caching proxy at: {os.environ.get("GDAL_HTTP_PROXY")}')
# Optional: Dask
cluster, client = initialize_dask(use_gateway=True, workers=(1,7), wait=False)
if cluster: display(cluster)
if cluster is None or 'LocalCluster' in str(type(cluster)): display(localcluster_dashboard(client))
Will use caching proxy at: easi-caching-proxy.caching-proxy:80 An existing cluster was found. Connecting to: easihub.1df2f465d26344ed9e723be1b41f5d75
VBox(children=(HTML(value='<h2>GatewayCluster</h2>'), HBox(children=(HTML(value='\n<div>\n<style scoped>\n …
Here are some example areas of interest. For simplicity they are defined with a bounding box. The pystac and odc-stac packages both accept a geopolygon as well.
# Vietnam - Ha Long
latitude = (20.5, 21.1)
longitude = (106.5, 107.2)
time=('2022-01-01', '2022-06-01')
# PNG Milne Bay
# latitude = (-10.8, -10)
# longitude = (149.7, 150.8)
# time=('2022-01-01', '2022-03-01')
# Fiji - blows up JHub memory due to antemeridian
# latitude = (-17.1, -16.2)
# longitude = (178.2, 180.0)
# time=('2020-02-01', '2020-02-20')
# west, south, east, north
bbox = [longitude[0], latitude[0], longitude[1], latitude[1]]
# Display bounding box on a map
display_map(longitude, latitude)
The odc-stac package There is an amount of detail in the following cell, which has been accumulated from various sources and our experience. Some
if do_landsat:
# STAC catalog and query
catalog = Client.open('https://landsatlook.usgs.gov/stac-server/')
product = 'landsat-c2l2-sr'
query_cfg = ["platform=LANDSAT_8", "landsat:collection_category=T1"]
# Search for available items
query = catalog.search(
collections=[product], datetime=f'{time[0]}/{time[1]}', bbox=bbox, query=query_cfg
)
items = list(query.get_items())
print(f"Found: {len(items):d} datasets")
# Rewrite URLs to use S3
def landsat_patch(uri: str) -> str:
"""Return the S3 version of the URI"""
return uri.replace('https://landsatlook.usgs.gov/data/', 's3://usgs-landsat/')
# Change or update STAC information for use by ODC
stac_cfg = json.load(open(f'{repo}/stac_cfgs/aws/{product}.json'))
# `stac_load` parameters
bands = ('blue', 'green', 'red', 'nir08', 'qa_pixel')
stac_call = {
'bands': bands, # Optional: selected bands
'bbox': bbox, # Bounding box. Also Geopolygon, GeoBox or None (full extent of items)
'chunks': {'x': 4096, 'y': 4096}, # Optional: if using Dask
'groupby': "solar_day", # "solar_day" = group scenes on same solar day into same time layer in cube
'stac_cfg': stac_cfg,
'patch_url': landsat_patch,
}
# Additional Landsat band specifications
band_specs = {
'red': {
'scale': 0.0000275,
'offset': -0.2
},
'nir08': {
'scale': 0.0000275,
'offset': -0.2
},
}
Found: 10 datasets
if do_sentinel:
# STAC catalog and query
catalog = Client.open('https://earth-search.aws.element84.com/v0')
product = 'sentinel-s2-l2a-cogs'
# Search for available items
query = catalog.search(
collections=[product], datetime=f'{time[0]}/{time[1]}', bbox=bbox,
)
items = list(query.get_items())
print(f"Found: {len(items):d} datasets")
# Rewrite URLs to use S3
def patch(uri: str) -> str:
"""Return the Sentinel-2 S3 version of the URI"""
return uri.replace('https://sentinel-cogs.s3.us-west-2.amazonaws.com/', 's3://sentinel-cogs/')
# Change or update STAC information for use by ODC
stac2odc_cfg = {
"sentinel-s2-l2a-cogs": {
"assets": {
"*": {"data_type": "uint16", "nodata": 0},
"SCL": {"data_type": "uint8", "nodata": 0},
"visual": {"data_type": "uint8", "nodata": 0},
},
"aliases": {"red": "B04", "green": "B03", "blue": "B02"},
},
"*": {"warnings": "ignore"},
}
# `stac_load` parameters
stac_call = {
'bands': ("B04",),
'crs': crs,
'resolution': 30,
# chunks={}, # <-- use Dask
# groupby="solar_day",
'stac_cfg': cfg,
'patch_url': patch,
}
# Optional: Explore the structure of a STAC item
display(items[0])
# Optional: List available band names and selected details
display(stac_landsat_assets_df(items[0]))
| ID: LC08_L2SP_126046_20220526_20220602_02_T1_SR |
| Bounding Box: [105.64938158691822, 19.178995478914924, 107.78826687413687, 21.272064615636843] |
| Datetime: 2022-05-26 03:17:33.812058+00:00 |
| datetime: 2022-05-26T03:17:33.812058Z |
| eo:cloud_cover: 19.23 |
| view:sun_azimuth: 83.57240769 |
| view:sun_elevation: 68.34626471 |
| platform: LANDSAT_8 |
| instruments: ['OLI', 'TIRS'] |
| view:off_nadir: 0 |
| landsat:cloud_cover_land: 2.21 |
| landsat:wrs_type: 2 |
| landsat:wrs_path: 126 |
| landsat:wrs_row: 046 |
| landsat:scene_id: LC81260462022146LGN00 |
| landsat:collection_category: T1 |
| landsat:collection_number: 02 |
| landsat:correction: L2SP |
| accuracy:geometric_x_bias: 0 |
| accuracy:geometric_y_bias: 0 |
| accuracy:geometric_x_stddev: 6.189 |
| accuracy:geometric_y_stddev: 6.305 |
| accuracy:geometric_rmse: 8.835 |
| proj:epsg: 32648 |
| proj:shape: [7721, 7551] |
| proj:transform: [30, 0, 566385, 0, -30, 2353215] |
| card4l:specification: SR |
| card4l:specification_version: 5.0 |
| created: 2022-07-01T14:42:55.362Z |
| updated: 2022-07-01T14:42:55.362Z |
| stac_extensions: ['https://landsat.usgs.gov/stac/landsat-extension/v1.1.1/schema.json', 'https://stac-extensions.github.io/view/v1.0.0/schema.json', 'https://stac-extensions.github.io/projection/v1.0.0/schema.json', 'https://stac-extensions.github.io/eo/v1.0.0/schema.json', 'https://stac-extensions.github.io/alternate-assets/v1.1.0/schema.json', 'https://stac-extensions.github.io/storage/v1.0.0/schema.json', 'https://stac-extensions.github.io/file/v1.0.0/schema.json', 'https://stac-extensions.github.io/accuracy/v1.0.0/schema.json', 'https://stac-extensions.github.io/card4l/v0.1.0/optical/schema.json', 'https://stac-extensions.github.io/classification/v1.0.0/schema.json'] |
| description: Landsat Collection 2 Level-2 Surface Reflectance Product |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_thumb_small.jpeg |
| Title: Thumbnail image |
| Media type: image/jpeg |
| Roles: ['thumbnail'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_thumb_small.jpeg'}} |
| file:checksum: 1340bda95ad26988235ffdaddce2f41c01fdcdaad1642e092068ca6b2cb6a2e5fe65e00fd33779d87d98f0845e1b31b34a4b3d02f6d9f9e5634d102f6b4db2dff4d2 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_thumb_large.jpeg |
| Title: Reduced resolution browse image |
| Media type: image/jpeg |
| Roles: ['overview'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_thumb_large.jpeg'}} |
| file:checksum: 13401f832454a9483a4e2917f9ddbaacb5b847d0c2d34d4118448b0eb16f8b9aea204f5c2bb755397999a4f6c9c07cc9a95470abbf2d022dfeb0246831ffefb47773 |
| href: https://landsatlook.usgs.gov/stac-browser/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1 |
| Title: HTML index page |
| Media type: text/html |
| Roles: ['metadata'] |
| Owner: |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.json |
| Title: Product Metadata File (json) |
| Description: Collection 2 Level-2 Product Metadata File (json) |
| Media type: application/json |
| Roles: ['metadata'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.json'}} |
| file:checksum: 1340b3c61387cb85a1cb473511a8cae59f9b38968264db7384828331abdefa93c1c98907b22639eb1761ae8c0b8b9520bc17355a2e76ca039fab4d1294778ad86e37 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B1.TIF |
| Title: Coastal/Aerosol Band (B1) |
| Description: Collection 2 Level-2 Coastal/Aerosol Band (B1) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data'] |
| Owner: |
| eo:bands: [{'name': 'B1', 'common_name': 'coastal', 'gsd': 30, 'center_wavelength': 0.44}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B1.TIF'}} |
| file:checksum: 13408dd035e9320f6003cca46659d953589a2e0f26706311c301b72a0924c723c2ef9398c7242d9a98eb37c5cd23dcae273ede12e0221b4375b4111983deb7970f20 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B2.TIF |
| Title: Blue Band (B2) |
| Description: Collection 2 Level-2 Blue Band (B2) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data'] |
| Owner: |
| eo:bands: [{'name': 'B2', 'common_name': 'blue', 'gsd': 30, 'center_wavelength': 0.48}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B2.TIF'}} |
| file:checksum: 13401254b195ab4d84dcf01b6b0aa5ac93a35a894d58e756121522c65f3bad6267f7f861b46125273bea8b159017f2e46b04b55b081451cde57983443179554d2050 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B3.TIF |
| Title: Green Band (B3) |
| Description: Collection 2 Level-2 Green Band (B3) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data'] |
| Owner: |
| eo:bands: [{'name': 'B3', 'common_name': 'green', 'gsd': 30, 'center_wavelength': 0.56}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B3.TIF'}} |
| file:checksum: 1340c067e56d9555c8e636b721f28ee7b27a88397eb8d089cade1f9f0d9f1ab7ff6f9dba54d40b4377d93c2501d05a4c6e8b82c3f072efa3172d48ed347f98d4fe1a |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B4.TIF |
| Title: Red Band (B4) |
| Description: Collection 2 Level-2 Red Band (B4) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data'] |
| Owner: |
| eo:bands: [{'name': 'B4', 'common_name': 'red', 'gsd': 30, 'center_wavelength': 0.65}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B4.TIF'}} |
| file:checksum: 134026c3003dfa291bf9087fa0a1a12cd90d3846945f376f2e6428a0a7d28b218a53ddf51f05648e3eb00bb18ebdf24ff60f07cc0749cc424e0ce395776053ff584f |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B5.TIF |
| Title: Near Infrared Band 0.8 (B5) |
| Description: Collection 2 Level-2 Near Infrared Band 0.8 (B5) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data', 'reflectance'] |
| Owner: |
| eo:bands: [{'name': 'B5', 'common_name': 'nir08', 'gsd': 30, 'center_wavelength': 0.86}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B5.TIF'}} |
| file:checksum: 1340492d9a474f2433ba33843ac6d2c07cb434ad654bb6ccf5ada438ad8d3a77188406f299cb98e2ea3d901c57fe3804253bd730556cb259e3fbc7524e00fdc6dfe9 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B6.TIF |
| Title: Short-wave Infrared Band 1.6 (B6) |
| Description: Collection 2 Level-2 Short-wave Infrared Band 1.6 (B6) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data', 'reflectance'] |
| Owner: |
| eo:bands: [{'name': 'B6', 'common_name': 'swir16', 'gsd': 30, 'center_wavelength': 1.6}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B6.TIF'}} |
| file:checksum: 134055ab13840958c43c1f97562b8859051dbe7c90e9915f218bf2deea292eca1572600f40820da938df1cec02dcac8e634aaf62bf2f84fd9302f6f428c4c51530f5 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B7.TIF |
| Title: Short-wave Infrared Band 2.2 (B7) |
| Description: Collection 2 Level-2 Short-wave Infrared Band 2.2 (B7) Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['data', 'reflectance'] |
| Owner: |
| eo:bands: [{'name': 'B7', 'common_name': 'swir22', 'gsd': 30, 'center_wavelength': 2.2}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_B7.TIF'}} |
| file:checksum: 1340c8326a22926bc1f067bb25d38425c61d00d7509d1ddc28485c491c40f3caad9940e355abb630f2fc89ec781c16c2b73d8f7b6b13447137b2fa9a114a26668f47 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_QA_AEROSOL.TIF |
| Title: Aerosol Quality Analysis Band |
| Description: Collection 2 Level-2 Aerosol Quality Analysis Band Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['metadata', 'data-mask', 'water-mask'] |
| Owner: |
| classification:bitfields: [{'name': 'fill', 'description': 'Corresponding pixels in L1 image bands are fill', 'offset': 0, 'length': 1, 'classes': [{'name': 'not_fill', 'description': 'L1 image band pixels are not fill', 'value': 0}, {'name': 'fill', 'description': 'L1 image band pixels are fill', 'value': 1}]}, {'name': 'retrieval', 'description': 'Valid aerosol retrieval', 'offset': 1, 'length': 1, 'classes': [{'name': 'not_valid', 'description': 'Aerosol retrieval is not valid', 'value': 0}, {'name': 'valid', 'description': 'Aerosol retrieval is valid', 'value': 1}]}, {'name': 'water', 'description': 'Water mask', 'offset': 2, 'length': 1, 'classes': [{'name': 'not_water', 'description': 'Not water', 'value': 0}, {'name': 'water', 'description': 'Water', 'value': 1}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 3, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 4, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'interpolated', 'description': 'Aerosol is interpolated', 'offset': 5, 'length': 1, 'classes': [{'name': 'not_interpolated', 'description': 'Aerosol is not interpolated', 'value': 0}, {'name': 'interpolated', 'description': 'Aerosol is interpolated', 'value': 1}]}, {'name': 'level', 'description': 'Aerosol level', 'offset': 6, 'length': 2, 'classes': [{'name': 'climatology', 'description': 'No aerosol correction applied', 'value': 0}, {'name': 'low', 'description': 'Low aerosol level', 'value': 1}, {'name': 'medium', 'description': 'Medium aerosol level', 'value': 2}, {'name': 'high', 'description': 'High aerosol level', 'value': 3}]}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_SR_QA_AEROSOL.TIF'}} |
| file:checksum: 13404ed026919ad17e262a193176a7d286149a1dbbc69c19f651693a32c2c84f723fb30e01d49e89b83fc0e12d83cfce8f2eaceeded83d520c07b83e88a2c9cf92ee |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_QA_PIXEL.TIF |
| Title: Pixel Quality Assessment Band |
| Description: Collection 2 Level-2 Pixel Quality Assessment Band Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['cloud', 'cloud-shadow', 'snow-ice', 'water-mask'] |
| Owner: |
| classification:bitfields: [{'name': 'fill', 'description': 'Corresponding pixels in L1 image bands are fill', 'offset': 0, 'length': 1, 'classes': [{'name': 'not_fill', 'description': 'L1 image band pixels are not fill', 'value': 0}, {'name': 'fill', 'description': 'L1 image band pixels are fill', 'value': 1}]}, {'name': 'dilated', 'description': 'Dilated cloud', 'offset': 1, 'length': 1, 'classes': [{'name': 'not_dilated', 'description': 'Cloud is not dilated or no cloud', 'value': 0}, {'name': 'dilated', 'description': 'Cloud dilation', 'value': 1}]}, {'name': 'cirrus', 'description': 'Cirrus mask', 'offset': 2, 'length': 1, 'classes': [{'name': 'not_cirrus', 'description': 'No confidence level set or low confidence cirrus', 'value': 0}, {'name': 'cirrus', 'description': 'High confidence cirrus', 'value': 1}]}, {'name': 'cloud', 'description': 'Cloud mask', 'offset': 3, 'length': 1, 'classes': [{'name': 'not_cloud', 'description': 'Cloud confidence is not high', 'value': 0}, {'name': 'cloud', 'description': 'High confidence cloud', 'value': 1}]}, {'name': 'shadow', 'description': 'Cloud shadow mask', 'offset': 4, 'length': 1, 'classes': [{'name': 'not_shadow', 'description': 'Cloud shadow confidence is not high', 'value': 0}, {'name': 'shadow', 'description': 'High confidence cloud shadow', 'value': 1}]}, {'name': 'snow', 'description': 'Snow/Ice mask', 'offset': 5, 'length': 1, 'classes': [{'name': 'not_snow', 'description': 'Snow/Ice confidence is not high', 'value': 0}, {'name': 'snow', 'description': 'High confidence snow cover', 'value': 1}]}, {'name': 'clear', 'description': 'Cloud or dilated cloud bits set', 'offset': 6, 'length': 1, 'classes': [{'name': 'not_clear', 'description': 'Cloud or dilated cloud bits are set', 'value': 0}, {'name': 'clear', 'description': 'Cloud and dilated cloud bits are not set', 'value': 1}]}, {'name': 'water', 'description': 'Water mask', 'offset': 7, 'length': 1, 'classes': [{'name': 'not_water', 'description': 'Land or cloud', 'value': 0}, {'name': 'water', 'description': 'Water', 'value': 1}]}, {'name': 'cloud_confidence', 'description': 'Cloud confidence levels', 'offset': 8, 'length': 2, 'classes': [{'name': 'not_set', 'description': 'No confidence level set', 'value': 0}, {'name': 'low', 'description': 'Low confidence cloud', 'value': 1}, {'name': 'medium', 'description': 'Medium confidence cloud', 'value': 2}, {'name': 'high', 'description': 'High confidence cloud', 'value': 3}]}, {'name': 'shadow_confidence', 'description': 'Cloud shadow confidence levels', 'offset': 10, 'length': 2, 'classes': [{'name': 'not_set', 'description': 'No confidence level set', 'value': 0}, {'name': 'low', 'description': 'Low confidence cloud shadow', 'value': 1}, {'name': 'reserved', 'description': 'Reserved - value not used', 'value': 2}, {'name': 'high', 'description': 'High confidence cloud shadow', 'value': 3}]}, {'name': 'snow_confidence', 'description': 'Snow/Ice confidence levels', 'offset': 12, 'length': 2, 'classes': [{'name': 'not_set', 'description': 'No confidence level set', 'value': 0}, {'name': 'low', 'description': 'Low confidence snow/ice', 'value': 1}, {'name': 'reserved', 'description': 'Reserved - value not used', 'value': 2}, {'name': 'high', 'description': 'High confidence snow/ice', 'value': 3}]}, {'name': 'cirrus_confidence', 'description': 'Cirrus confidence levels', 'offset': 14, 'length': 2, 'classes': [{'name': 'not_set', 'description': 'No confidence level set', 'value': 0}, {'name': 'low', 'description': 'Low confidence cirrus', 'value': 1}, {'name': 'reserved', 'description': 'Reserved - value not used', 'value': 2}, {'name': 'high', 'description': 'High confidence cirrus', 'value': 3}]}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_QA_PIXEL.TIF'}} |
| file:checksum: 13406a1438b6ee558b4c14a53e602e353b08b958128c9e86bdb30ff6854ba7f2261164977e19fe635ff550f992b2127a39a4fb6b4dff4a4420a55715be3fb4f0b25b |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_QA_RADSAT.TIF |
| Title: Radiometric Saturation Quality Assessment Band |
| Description: Collection 2 Level-2 Radiometric Saturation Quality Assessment Band Surface Reflectance |
| Media type: image/vnd.stac.geotiff; cloud-optimized=true |
| Roles: ['saturation'] |
| Owner: |
| classification:bitfields: [{'name': 'band1', 'description': 'Band 1 radiometric saturation', 'offset': 0, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 1 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 1 is saturated', 'value': 1}]}, {'name': 'band2', 'description': 'Band 2 radiometric saturation', 'offset': 1, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 2 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 2 is saturated', 'value': 1}]}, {'name': 'band3', 'description': 'Band 3 radiometric saturation', 'offset': 2, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 3 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 3 is saturated', 'value': 1}]}, {'name': 'band4', 'description': 'Band 4 radiometric saturation', 'offset': 3, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 4 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 4 is saturated', 'value': 1}]}, {'name': 'band5', 'description': 'Band 5 radiometric saturation', 'offset': 4, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 5 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 5 is saturated', 'value': 1}]}, {'name': 'band6', 'description': 'Band 6 radiometric saturation', 'offset': 5, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 6 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 6 is saturated', 'value': 1}]}, {'name': 'band7', 'description': 'Band 7 radiometric saturation', 'offset': 6, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 7 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 7 is saturated', 'value': 1}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 7, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'band9', 'description': 'Band 9 radiometric saturation', 'offset': 8, 'length': 1, 'classes': [{'name': 'not_saturated', 'description': 'Band 9 is not saturated', 'value': 0}, {'name': 'saturated', 'description': 'Band 9 is saturated', 'value': 1}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 9, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 10, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'occlusion', 'description': 'Terrain not visible from sensor due to intervening terrain', 'offset': 11, 'length': 1, 'classes': [{'name': 'not_occluded', 'description': 'Terrain is not occluded', 'value': 0}, {'name': 'occluded', 'description': 'Terrain is occluded', 'value': 1}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 12, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 13, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 14, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}, {'name': 'unused', 'description': 'Unused bit', 'offset': 15, 'length': 1, 'classes': [{'name': 'unused', 'description': 'Unused bit', 'value': 0}]}] |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_QA_RADSAT.TIF'}} |
| file:checksum: 13406086807a73876666e1d669a43566a3541349826d05ba1a4b12b84dfda39718d9927e8d0b42d548fcec8d92069cd98a943cc3b775c3c45f22f324fa803dae7b86 |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_ANG.txt |
| Title: Angle Coefficients File |
| Description: Collection 2 Level-2 Angle Coefficients File (ANG) |
| Media type: text/plain |
| Roles: ['metadata'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_ANG.txt'}} |
| file:checksum: 1340435e8925d738971258a09f30cba120d967e6b27de2ecf034da7812f357ac1c6d01ac9acc1809558c4a31d2ae10629e8db262a6d28e5a75d3f4e31e16cb51502a |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.txt |
| Title: Product Metadata File |
| Description: Collection 2 Level-2 Product Metadata File (MTL) |
| Media type: text/plain |
| Roles: ['metadata'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.txt'}} |
| file:checksum: 1340fccf985d9f86641fbc0f25acd8ef10f83e8e3d8bbaa8704a0a0f9923429bbfc9ec731a82d1321074f33ecb967c3b9872213880d5331c89843e4607c8b8b8b9bf |
| href: https://landsatlook.usgs.gov/data/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.xml |
| Title: Product Metadata File (xml) |
| Description: Collection 2 Level-2 Product Metadata File (xml) |
| Media type: application/xml |
| Roles: ['metadata'] |
| Owner: |
| alternate: {'s3': {'storage:platform': 'AWS', 'storage:requester_pays': True, 'href': 's3://usgs-landsat/collection02/level-2/standard/oli-tirs/2022/126/046/LC08_L2SP_126046_20220526_20220602_02_T1/LC08_L2SP_126046_20220526_20220602_02_T1_MTL.xml'}} |
| file:checksum: 134015994eb70c25dd499e25123e6bcb38edde0717259db95020c00dbdcc43a3c129b2ded4d4af366ead468b18bcbb2286df21d379d2feda5b0d85e4ea8b278056fd |
| Rel: self |
| Target: https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr/items/LC08_L2SP_126046_20220526_20220602_02_T1_SR |
| Rel: parent |
| Target: https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr |
| Rel: collection |
| Target: https://landsatlook.usgs.gov/stac-server/collections/landsat-c2l2-sr |
STAC API
| Rel: root |
| Target: |
| Media Type: application/json |
| title | description | eo:bands | classification:bitfields | |
|---|---|---|---|---|
| coastal | Coastal/Aerosol Band (B1) | Collection 2 Level-2 Coastal/Aerosol Band (B1)... | [{'name': 'B1', 'common_name': 'coastal', 'gsd... | NaN |
| blue | Blue Band (B2) | Collection 2 Level-2 Blue Band (B2) Surface Re... | [{'name': 'B2', 'common_name': 'blue', 'gsd': ... | NaN |
| green | Green Band (B3) | Collection 2 Level-2 Green Band (B3) Surface R... | [{'name': 'B3', 'common_name': 'green', 'gsd':... | NaN |
| red | Red Band (B4) | Collection 2 Level-2 Red Band (B4) Surface Ref... | [{'name': 'B4', 'common_name': 'red', 'gsd': 3... | NaN |
| nir08 | Near Infrared Band 0.8 (B5) | Collection 2 Level-2 Near Infrared Band 0.8 (B... | [{'name': 'B5', 'common_name': 'nir08', 'gsd':... | NaN |
| swir16 | Short-wave Infrared Band 1.6 (B6) | Collection 2 Level-2 Short-wave Infrared Band ... | [{'name': 'B6', 'common_name': 'swir16', 'gsd'... | NaN |
| swir22 | Short-wave Infrared Band 2.2 (B7) | Collection 2 Level-2 Short-wave Infrared Band ... | [{'name': 'B7', 'common_name': 'swir22', 'gsd'... | NaN |
| qa_aerosol | Aerosol Quality Analysis Band | Collection 2 Level-2 Aerosol Quality Analysis ... | NaN | [{'name': 'fill', 'description': 'Correspondin... |
| qa_pixel | Pixel Quality Assessment Band | Collection 2 Level-2 Pixel Quality Assessment ... | NaN | [{'name': 'fill', 'description': 'Correspondin... |
| qa_radsat | Radiometric Saturation Quality Assessment Band | Collection 2 Level-2 Radiometric Saturation Qu... | NaN | [{'name': 'band1', 'description': 'Band 1 radi... |
xarray cube¶xx = stac_load(items, **stac_call)
heading(xarray_object_size(xx))
display(xx)
display(xx.odc.geobox)
aspect = image_aspect(xx)
<xarray.Dataset>
Dimensions: (y: 2243, x: 2455, time: 6)
Coordinates:
* y (y) float64 2.335e+06 2.335e+06 ... 2.268e+06 2.268e+06
* x (x) float64 6.558e+05 6.558e+05 ... 7.294e+05 7.294e+05
spatial_ref int32 32648
* time (time) datetime64[ns] 2022-01-02T03:17:29.057704 ... 2022-05...
Data variables:
blue (time, y, x) uint16 dask.array<chunksize=(1, 2243, 2455), meta=np.ndarray>
green (time, y, x) uint16 dask.array<chunksize=(1, 2243, 2455), meta=np.ndarray>
red (time, y, x) uint16 dask.array<chunksize=(1, 2243, 2455), meta=np.ndarray>
nir08 (time, y, x) uint16 dask.array<chunksize=(1, 2243, 2455), meta=np.ndarray>
qa_pixel (time, y, x) uint16 dask.array<chunksize=(1, 2243, 2455), meta=np.ndarray>Following Cloud_and_pixel_quality_masking.ipynb, we choose the "cloud mask filtered" method to remove false-positive cloud features before further analysis.
Note that there are some differences in the key or value labels between the Landsat flags definition in the referenced notebook and available from STAC, which is just due to human interpretation. See the Landsat product links at the top of this notebook for further information.
# Convert pixel quality flag descriptions to ODC format
flags_def = stac_landsat_flags_to_dc(
items[0].assets['qa_pixel'].to_dict().get('classification:bitfields')
)
# heading('"qa_pixel" flags definition')
# display(flags_def)
# Cloud mask flags (logical AND?)
quality_flags = {
'cloud': 'cloud', # True where there is cloud
'cirrus': 'cirrus', # True where there is cirrus cloud
'shadow': 'shadow', # True where there is cloud shadow
}
# Set bit mask: True=cloud, False=non-cloud
mask, _= masking.create_mask_value(flags_def, **quality_flags)
# Add the cloud mask to our dataset
xx['cloud_mask'] = (xx['qa_pixel'] & mask) != 0 # bitwise-and != 0 simulates an 'or'
# Apply morphological processing on the cloud mask
filters = [("opening", 2),("dilation", 2)]
xx['cloud_mask_filtered'] = mask_cleanup(xx['cloud_mask'], mask_filters=filters)
# Apply the the cloud-mask to the data variables
clear_filtered = erase_bad(xx.drop_vars(['cloud_mask_filtered', 'cloud_mask', 'qa_pixel']),
xx['cloud_mask_filtered'])
# Apply scale and offset
scale = stac_cfg[product]['assets']['*']['scale']
offset = stac_cfg[product]['assets']['*']['offset']
clear_filtered = (clear_filtered * scale + offset).astype(np.float32)
If using dask then the above calculations may have been queued but not yet calculated. Dask will run the calculations when required, for example when creating an image or writing to a file. In this case, use persist() to force the calulations.
xx = xx.persist()
clear_filtered = clear_filtered.persist()
# Optional: Check the data structures and sizes
# heading(xarray_object_size(xx))
# display(xx)
# heading(xarray_object_size(clear_filtered))
# display(clear_filtered)
These simpler "matplotlib backend" plot functions (rgb, xx.plot) render images, from the full data arrays, on the Jupyter notebook server. This may exceed the available Jupyter kernel memory depending on the size of the area of interest (size of the dataset). As such these methods are best suited to demonstrating smaller areas of interest.
To visualise larger areas of interest consider either subsampling the data arrays (xarray indexing) or try the Holoviz stack:
# Optional: Plot and RGB timeseries
rgb(xx, ['red', 'green', 'blue'], col='time', col_wrap=4)
# Optional: plot the masked and scaled data
rgb(clear_filtered, ['red', 'green', 'blue'], col='time', col_wrap=4)
2022-09-29 08:52:37,154 - distributed.client - ERROR - Failed to reconnect to scheduler after 30.00 seconds, closing client
Its good practice to shutdown the dask cluster when the work is complete (no further work on the loaded and processed data).
# client.close()
# if cluster: cluster.close()